Game monitoring

ABSTRACT

The present invention relates to a device, a system and a method for tracking at least one object and player on a playing field/court and/or for monitoring the course of the game.

BRIEF SUMMARY OF THE INVENTION

Society without sporting activities for physical fitness is virtuallyunimaginable. However, many sports persons want not just to participatein sport, but to get better in their corresponding type of sport, inorder to be competitive in competition events. In this case, in manytypes of sport the lack of meaningful data regarding a person's ownperformance poses a major problem since without such data mistakes andpotentials for improvement can be found only with difficulty.

Activity monitoring systems, for example for ball sports such as soccer,are known, and comprise for example sensors that are incorporated intothe ball and recognize the movements thereof. Said sensors can ascertainthe trajectory and/or the position of the ball for example with the aidof a magnetic field, as explained in EP 2 657 924 A1 for example.

Further monitoring and/or information systems in the area of ball sportsor racket sports are so-called smartcourt systems. However, tracking themovement of the usually fast objects on the playing field/court is verydifficult and complicated, as a result of which in many cases complexsystems are used which enable tracking of the object on the playingfield/court and in 3D space. As a result, inter alia decisions in thegame, such as “line calls” in tennis, for example, can be made in realtime or statistics regarding the course of a sports event can becompiled. These complex systems often comprise a plurality of camerasand/or movable cameras that enable the tracking of an object, such asthe ball, for example.

The smartcourt system in WO 2013/124856 A1 for example comprises amultiplicity of video cameras installed with rigid positioning. Theimage information from the different camera perspectives is used togenerate a 3D model of the court, the ball and the players. On the basisof the movements captured therein, conclusions are drawn regarding thetype of stroke, the ball speed and the hit points and can be viewed bythe sports persons.

U.S. Pat. No. 9,737,784B1 concerns an electronic officiating system inwhich a plurality of camera arrays are mounted at different heights onthe net post. Objects can be tracked since they pass through the fieldsof view of the individual arrays at different points in time, from whichconclusions can be drawn regarding the movements of said objects.

WO 2017/100465 A1 discloses a system comprising a camera arranged inproximity to the post, which camera covers both sides of the court. Inthis case, a court map including the positioning of the court, of thenet and/or of the camera is created by means of an autocalibrationprocess. This enables the 2D coordinates from the video image generatedduring the match to be converted into 3D coordinates in the court map.

Current smartcourt solutions for ball games and/or racket games aretherefore complex and expensive, as a result of which sports persons whodo not train or play as professionals have no opportunity to analyzetheir games. Furthermore, systems with one camera perspective do notafford the possibility of analyzing types of strokes, for example intennis. Moreover, existing systems are not able to ascertain the correctscore if there is a difference between how the system captures thecourse of the game and the actual decisions of the players.

Therefore, the object of the present invention is to provide a system, amethod and a computer program product that overcome the disadvantagesmentioned above and give even amateur players an opportunity to monitorand analyze their games.

The present invention therefore relates to a method for game monitoringcomprising the following steps:

a) determining the playing field/court;

b) recognizing at least one person on the playing field/court;

c) identifying objects on the playing field/court;

d) determining at least one trajectory;

wherein at least one event is identified by means of which the scorescalculated in the course of the game are corrected by means ofevent-based result ascertainment (backward scoring) and the real finalscore of the game is output. In one embodiment, determining the playingfield/court is effected by means of calibration, wherein individualpoints in a two-dimensional (2D)-image are assigned to a specific pointin a two-dimensional (2D) model of the playing field/court. In onepreferred embodiment, the calibration is effected by way of a homographymatrix in which at least four points are used which correlate in theimage and the model. In one particular preferred embodiment, these arefour corner points of the playing field/court.

In one embodiment, recognizing persons on the playing field/courtcomprises the following steps:

-   -   a) generating at least one background image of a playing        field/court;    -   b) generating a background model;    -   c) separating each background image from each currently recorded        image which comprises a person by:        -   i) determining an absolute difference between the background            model and the currently recorded image, and        -   ii) generating a foreground mask,            wherein the foreground mask is characterized in that all            pixels belonging to the foreground have the value 255 and            the pixels of the background have the value 0. In one            preferred embodiment, pixels for which the difference is            greater than a defined limit value are determined as            foreground and pixels with a smaller difference are            determined as background.

In a further embodiment of the present invention, the object on theplaying field/court is identified by means of movements of each recordedimage. In one preferred embodiment, determining the movement comprisesthe following steps:

-   a) creating difference images in respect of successive images;-   b) calculating absolute difference images, in which a movement is    visible at two places in the difference image, it preferably being a    first position, at which the object was situated in the previous    image, and a second position, at which the object is situated in the    current image;-   c) generating a first movement mask for the one first difference    image and a second movement mask for the one second difference    image; and-   d) generating a final movement mask comprising the movements in the    one first movement mask and the one second movement mask.

In one particularly preferred embodiment, the individual pixels of bothmovement masks are added by a bitwise AND operator.

In one embodiment, the trajectory path or the trajectory of the objectis ascertained by the following steps being carried out:

-   a) fixing a first and a second starting point of a potential    trajectory;-   b) combining the potentially fixed starting points with all    potential objects of the temporally following image;-   c) checking each object as to whether an already existing trajectory    is continued;-   d) ascertaining positions of the object in successive images;-   e) comparing the trajectories from a) and d);-   f) determining a trajectory course; and-   g) classifying the trajectory by determining a second trajectory    following the trajectory.

In one preferred embodiment, pixels of objects which correlate withpixels of persons are not taken into account.

In a further embodiment, identifying events in the course of the gamecomprises the following steps:

-   a) capturing a trigger event;-   b) comparing the position of at least one person and one object with    the playing field/court model;-   c) correlating with a database.

In one preferred embodiment, the events have been captured in acategorized data set in a previous step in order to carry out thecorrelating with the aid of an artificial intelligence trained with thisdata set.

In one embodiment of the present invention, correcting the scorecalculated in the course of the game comprises the following steps:

-   a) monitoring the possible scores during the game;-   b) checking the compatibility of all possible scores with the    detected events;-   c) checking the plausibility of the remaining scores.

The present invention furthermore relates to a device for tracking atleast one object and player on a playing field/court comprising:

-   a) an image recording system comprising    -   i) at least one camera arranged in such a way that the entire        playing field/court is captured;    -   ii) at least one transmission module designed to store the data        recorded by the camera and/or to transmit them to a computing        unit, a storage means, or system;        wherein the at least one camera and the at least one        transmission module are attached to at least one net post;-   b) a computing unit, comprising modules for    -   iii) playing field/court ascertainment;    -   iv) person recognition;    -   v) object determination;    -   vi) trajectory recognition;    -   vii) event identification; and    -   viii) event-based result ascertainment.

Furthermore, the present invention relates to a computer program forcarrying out a method described above when the computer program isexecuted on a computer.

The present invention also relates to a computer program product

-   a) comprising a storage medium, on which a computer program    described above is stored; and/or-   b) which can be loaded directly into the internal memory of a    digital computer and comprises software code sections that enable    the steps in accordance with the method described above to be    carried out when the product runs on a computer.

The object is furthermore characterized by the embodiments in the claimsand described in greater detail by the explanations in the description,the examples and the drawings.

DRAWINGS

FIG. 1 A schematic illustration of a homography transformation withinthe field/court recognition

FIG. 2 A schematic illustration of trajectory ascertainment

a) difference image t; b) difference image t+1; c) hit point; d)difference image t+2; e) difference image t+3; f) stroke

FIG. 3 A schematic illustration of a program sequence for a tennis match

FIG. 4 Schematic checking for compatibility in the course of the game bymeans of a scoring tree

DETAILED EXPLANATION OF THE INVENTION

The present invention relates to a method, a device, a computer programand a computer program product for tracking an object on a staticplaying field/court. The object can be any conceivable object in a game,such as, for example, a handball, a soccer ball, a tennis ball, ashuttlecock or the like. In one preferred embodiment of the presentinvention, a ball used in a racket sport is involved. In oneparticularly preferred embodiment, the ball game is tennis.

Hereinafter, in order to simplify the description, the invention isexplained with regard to a game of tennis encompassing a tennis court, atennis ball, a tennis player and a tennis racket. However, this isrepresentative of other types of sport encompassing a playingfield/court, at least one player and an object, such as a ball, forexample. Furthermore, the use of “a/an” or “one” is used to describe oneor more elements, components, steps, modules and/or things. These termsshould be interpreted such that they encompass one or at least one andthe singular also encompasses the plural, unless the contrary isobviously meant.

The invention relates to a method for tracking a person and/or object ona playing field/court. In this case, the method comprises the steps offield/court recognition, person recognition, object recognition,trajectory recognition, event recognition and backward scoring. Thesesteps are explained in greater detail below and are carried out onsuitable means, such as a data processing system or a computer, forexample.

The field/court recognition, i.e. the determination or initialization ofthe playing field/court, serves for calibration in order that theinformation from two-dimensional (2D) image recordings can be projectedinto another two-dimensional (2D) image, the playing field/court model.Said information comprises for example hit points and/or whereabouts ofat least one player. In one preferred embodiment of the presentinvention, the calibration is effected by way of a homography matrix.The latter describes the translation, rotation and distortion of one andthe same planar surface, e.g. the playing field/court, in two arbitraryimages. The homography matrix is calculated by using preferably at leastfour points from the image recording which correlate with four points inthe model image. Preferably, use is made of the four corner points ofthe individual field/court, for example in tennis, which either wereselected beforehand by the user or were calculated by means of imageprocessing methods. Customary methods of image processing or frommachine learning can be involved here. One example from image processingwould be e.g. edge detection, in which demarcations between differentareas (e.g. line on a playing field/court) can be recognized within animage with the aid of various mathematical operators (edge filters) suchas e.g. a Laplacian filter, a Sobel operator, a Scharr operator. Thecorner points of the playing field/court can then be calculated bycalculating the points of intersection of recognized lines. Oneexemplary method from machine learning would be a convolutional neuralnetwork (CNN), for which a previously classified data set must in turnbe available. A convolutional neural network is a network of neuronswhich is based on the biological functioning of a brain and isimplemented at the machine level. In this case, localized features areextracted from input images and the latter convolve image fields bymeans of filters. The input to a convolutional layer is an m x m×rimage, wherein m is the height and width of the image and r is thenumber of channels. By way of example, an RGB image has r=3 channels.These data are transferred by a plurality of layers and repeatedlyrefiltered and undersampled.

In a further step of the present invention, persons situated on theplaying field/court are identified. The terms players, participants,persons and/or humans are used interchangeably in the description anddenote all humans or non-humans, such as animals, situated on theplaying field/court. This is in contrast to the objects, which areinanimate objects. The recognition of at least one player or otherparticipant is effected with the aid of a so-called backgroundsubtraction method. This subtractor serves for separating objects in theforeground from objects in the background of an image and for checkingthe objects with regard to their shape. In order to create a backgroundmodel for the background subtraction, at least one image of the emptyplaying field/court is recorded. The further step involves using thisbackground image, i.e. the empty playing field/court, to effectinitialization using the known “Mixture of Gaussians BackgroundSubtractor” and calculating a background model. This technique assumesthat the intensity values of each pixel in the video can be modelled bymeans of the “Mixture of Gaussians” method. A simple heuristicdetermines which intensities are most likely to be in the background.The pixels which do not correspond to them then become the foregroundpixels. Preferably, for calculating and modelling the background, aplurality of images recorded by the image system are used to update thebackground. In one preferred embodiment, an update is effected afterevery 20 recorded image data, wherein continuous differences between thecurrent image and the background model are incorporated into thebackground model. This ensures that constant changes in the backgroundare not detected as foreground and thus potentially as humans.

For person recognition, in one embodiment of the present invention, aseparation of the foreground image and background image is carried outfor each image generated by the image recording system. This gives riseto a foreground mask, which is particularly preferably characterized inthat all pixels belonging to the foreground have the value 255 and thepixels of the background have the value 0. In order to achieve this, theabsolute difference between the background model and the currentlyrecorded image of the image recording system is calculated by acomputer. The use of the absolute difference allows the differencesbetween two images to be reproduced, wherein the absolute differenceindicates the absolute value of the difference between two images. Inone preferred embodiment, pixels for which the difference is greaterthan a defined limit value are recorded as foreground. In this case,pixels with a smaller difference are regarded as background. Theextracted objects from the foreground mask are potential humans. Bymeans of a collection of data contained on the computer or some othermedium and/or correlating with data on the Internet, the objects areexamined and/or compared on the basis of their height, width, areaand/or position in the image and are classified as human or non-human.

In the following method, the homography matrix generated in thefield/court recognition and the bottommost pixel of a human ornon-human, said pixel being classified in the object recognition, areused to ascertain the position at which the object is situated in the 2Dmodel of the playing field.

Besides object recognition, in a further step of the present invention,objects on the playing field/court are detected by means of the imagerecording system. The term “objects” is used interchangeably with theterms “ball” or “ball object” and encompasses all suitable objects whichare used in the same way as a ball, for example in games that involvereturning the ball, games that involve scoring goals and/or games thatinvolve striking the ball. The object here does not require a specificshape, but rather is dependent on its suitability. Objects or ballsencompass but are not restricted to the following listing: tennis balls,shuttlecocks, flying disks, soccer balls, volleyballs, inter alia.Accordingly, the ball can be round, oval, disk-shaped or ring-shaped,equipped with grips or flight stabilizers (feathers), braided,perforated, provided with a movable filling, hollow or solid.

In order to detect the movement of the balls, movements in images of theimage recording system are examined. This is done by creating differenceimages in respect of successive images from the image recording system,which makes it possible to detect movement-dictated changes between theimages. In order to be able to recognize the object against differentbackgrounds, one embodiment involves creating absolute difference imagesbetween successive images. When the ball moves through the field of viewof the image recording system, it moves past different backgrounds,which may be brighter, such as lamps, sun, for example, or darker, suchas floor or walls, for example, than the ball. With the aid of theabsolute differences, a movement is always made visible at two places inthe difference image. These places are preferably a first position, atwhich the ball was situated in the previous image, and a secondposition, at which the ball is situated in the current image.Accordingly, the disappearance of the ball on a first image and itsappearance are detected.

One embodiment involves preferably capturing the exclusive movement on acurrent image at the point in time t=1 (image 1 hereinafter). For thispurpose, firstly all movements, i.e. the appearance and thedisappearance of the ball, are ascertained. The ascertaining is effectedin a first step by generating a difference image between an initialimage (image 0 hereinafter) and image 1. The disappearance of the ballfrom a first position in image 0, at which the ball was situated, andthe appearance of the ball in image 1 in a second position, at which theball is situated, are recognized as a result. A further embodimentinvolves generating a second difference image (difference image 2)between image 1 and a second image (image 2 hereinafter). Thedifferential image 2 detects the disappearance of the ball from image 1to image 2 and the appearance of the ball on image 2.

In a further step of object recognition, movement masks are generatedfor one first and the one second difference image, said movement masksbeing identified hereinafter as movement mask 1 and movement mask 2. Inthese movement masks, all pixels for which the value of the absolutedifferences lies above a defined threshold value acquire the value 255.All pixels that lie below the defined threshold value acquire the value0. In this case, a predefined value, such as a predefined thresholdvalue, for example, is a fixed value and/or a value that is determinedat an arbitrary point in time before a calculation that compares aspecific value with the predefined value is carried out. Accordingly,movements in the movement masks and the pixels corresponding to themhave the value 255. In the following step, a final movement mask(movement mask 3) is generated. This final movement mask 3 contains themovements that appear in the one first movement mask 1 and in the onesecond movement mask 2. In one preferred embodiment of the presentinvention, the final movement mask is created using a logic adding ofthe individual pixels of both movement masks, preferably by means of abitwise AND operator. Such a final movement mask 3 makes it possible toextract potential objects, such as balls. In one preferred embodiment,for the extracted ball objects a decision is taken as to whether theyare situated within or outside a recognized human. This has a particularimportance in trajectory recognition, in particular, since the movementof potential objects within a recognized human is usually effected bythe human and not the object, for example the ball.

In a further embodiment of the present invention, the trajectory or thetrajectory of the object is ascertained. It is assumed here that theobject can start a trajectory and also continue a trajectory. Allpotential objects detected as not within a recognized human by the imagerecording system and/or the image processing system are fixed here aspotential starting points of a new trajectory. In a next step, thesepotentially fixed starting points are combined with all potentialobjects of the temporally following image.

By way of example, the ball objects A and B recorded in a first imageare ascertained. Accordingly, the associated trajectories 1(A) and 2(B)are generated. The two potential ball objects C and D are ascertained ina second image. On the basis of the trajectories generated in the secondimage, the following trajectories are generated: 1(A, C), 2(A, D), 3(B,C), 4(B, D). In one particularly preferred embodiment, two furthertrajectories 5(C) and 6(D) are generated since, as described above, eachobject is permitted to start a new trajectory.

In a following step for recognizing the trajectory, each potential ballobject is checked as to whether it continues an already existingtrajectory with two or more ball objects. In order to carry out suchchecking of the trajectory trajectories, positions of the ball onsuccessive images are ascertained and compared with the existingtrajectory that was defined by means of one first and one secondstarting point. The checking is preferably additionally effected usingdata concerning the flight direction of the ball in space and/or theacceleration thereof. These data, i.e. the flight direction andacceleration, can be calculated by the juxtaposition of three objectpoints and thus make it possible to predict the position of thefollowing ball object in the image. Each ball situated in the vicinityof the predicted position ascertained is added here to the calculatedtrajectory until the latter has a starting point and an end point.

In one particularly preferred embodiment, ball objects which areascertained within the human, i.e. the pixels of which correlate withthe pixels of a human, are not used to create new trajectories, since,as already explained above, a movement situated there may, under certaincircumstances, be caused by the human himself/herself and the latter'smovement, and not by the object, that is to say the ball itself.However, ball objects in such positions, i.e. in the correlative regionof recognized humans, can be used when ascertaining the trajectory, i.e.for the trajectory continuation and the prediction thereof.

The creation of new trajectories for each potential ball objectidentified results in numerous potential trajectories, among whichnumerous trajectories are present multiplied. In order to prevent thisredundancy, in one embodiment of the present invention, a check iscarried out after each processed image. In the course of such a check,identical or similar trajectories are ascertained, recognized and thendeleted from the computational data. In one preferred embodiment of thepresent invention, trajectories with fewer than eight ascertained ballpositions, preferably with fewer than six, particularly preferably withfewer than four, ascertained ball positions, are additionally deletedfrom the computational data as well. Deleting the data sets with a smallnumber of actually ascertained ball positions reduces noise which, undercertain circumstances, could result in corrupted actual trajectoryascertainment.

After complete detection of a plurality of trajectories, which maydiffer in terms of their type, said trajectories are classified in oneembodiment of the present invention. The classification is effecteddepending on the type of sport. In this case, preferably data sets oftrajectory paths and trajectories that are customary in the type ofsport are stored and compared with the ascertained trajectories from theimage recording system. In the example of the sport of tennis, thetrajectories are classified into different classes such as stroke,serve, hit point, over net, end of rally, etc. The terms“classification” or “characterization” are used interchangeably here andin this case describe the identification of a trajectory according totype, trajectory and optionally event, such as, for example, the serve,a penalty kick or a free throw. A trajectory (K1 hereinafter) isclassified by a trajectory that succeeds it being determined. In orderto determine the trajectory following K1, at least one trajectoryascertained after K1 is checked and the trajectory determined as anassociated successor trajectory (K2) is checked, preferably alltrajectories following K1 are checked as a potential associatedsuccessor trajectory (K2′) of K1. In one preferred embodiment, alltrajectories (K2′) are checked which begin at a later point in time, butfewer than 30 images later. In one particularly preferred embodiment,the trajectory which in spatial and temporal combination is at thesmallest distance from K1 is defined as an associated successortrajectory K2.

In one embodiment, an associated predecessor trajectory (K1) isadditionally ascertained for the successor trajectory (K2) under thesame criteria as explained above for the successor trajectory K2. Such acorrelation of the trajectories prevents trajectories generated on otherplaying fields/courts from being characterized as a trajectory of thecorrect playing field/court.

After a corresponding correlation of trajectories K1 and K2, i.e. K1 wasdetermined as a predecessor for K2, and conversely K2 was determined asa successor for K1, for the following characterization, directions,direction changes and/or angles between the trajectories are capturedand compared with defined patterns from a database. A correspondingdatabase here can be the combination of dedicated sets of data which arestored internally (locally) on the system or are established externallyon storage media. Furthermore, such a database can also be retrieved ina cloud-based storage means, in a network or on the Internet.

In one embodiment of the present invention, the trajectories ascertainedand determined by the image recording system and processing system areused for defining events in the game. In this case, individual events inthe entire course of the game are determined and classified. The eventsin the course of the game encompass but are not limited to the followingenumeration: beginning and end of each rally or of a game, change ofserving end or players' change of ends, end of set and beginning of atiebreak, free kick, penalty kick, free throw, pitch, strike, fowl,inter alia. As already explained above, the event recognition will beexplained on the basis of a game of tennis for simplification. Theevents are also referred to as triggering events. These triggeringevents are stored within a database and categorized within the latter.

The beginning of a rally is characterized by one player serving, forwhich reason event recognition waits for a trajectory that wascharacterized as a serve. When this serve trajectory occurs, firstly acheck is made, by way of the standing position of the associated player,as to whether the serve was detected on the correct place on the tenniscourt and then where the two players were situated at the time of theserve. Such position recognition for the players is effected here withthe aid of the image recording system and the object recognition alreadydescribed above. In a further embodiment, this local information of theplayers subsequently provides information, with regard to the score, asto whether the current score can be associated with even or odd gamesand for the statistics as to whether a second serve is involved (twoserves owing to the same position). After such a rally start has beenrecognized, further events in the game are ascertained in one preferredembodiment. Said further events may be for example a hit point after aserve. Such a hit point in tennis must be situated in a different partof the court depending on the position of the serving player. This hitpoint or this event provides information about whether it is probablethat a second serve will follow. In one preferred embodiment, for eachevent ascertained, a check is made as to whether this event occurred onthe correct playing field/court and/or whether it occurred within theplaying field/court, i.e. the playing field/court lines. Such anascertainment as to whether the event took place within or outside theplaying field/court region, the ball and/or player position iscorrelated with the field/court data ascertained. In one preferredembodiment, the distance between the recognized hit point and the linesdelimiting the playing field/court is additionally taken as a basis foracquiring a probability with which the decision of the system was takencorrectly. The closer the hit point is to the lines, the lower theprobability in this case.

In one embodiment of the present invention, the events and position ofthe players and balls are recognized in the playing field/court mode,wherein the hit points and standing positions are transformed into theplane of the tennis court model with the aid of the homography matrixdetermined in the calibration. Events such as, for example, balls flyingover the net and the end of the rally preferably also undergo a check toascertain whether the associated player is situated in the correct zone.Furthermore, in one preferred embodiment, strokes are characterizedaccording to a data set. This data set for strokes can comprise generallocal databases, external information from networks or the Internet or aself-generating collection of data. Preferably, the data set for strokesis a collection of data which has been extracted from previousrecordings and categorized. The data can be extracted both to localstorage media and to storage media in local networks or on the Internet.

Such a collection of data additionally stores events, such as a changeof serving end, for example, which always takes place at the beginningof a new game and constitutes a game-ending event. In addition, thisevent can be confirmed by virtue of the fact that a player who did notperform the previous serves performs a plurality of serves successively.These game-influencing events are also referred to as triggering events.The same applies to changes of ends, which likewise constitute agame-ending event. In addition, in one embodiment of the presentinvention, all events can be identified and thereby characterized by aplayer or a third person independently by inputting. Such inputting bythe user can lead, in an automated manner, to an updating of thecollections of data stored in the database. Furthermore, in a furtherembodiment, elements can be provided which enable manual inputting of anevent, such as the end of a match, for example. By way of example,remote controls, smartphones, buttons, touchscreens or other devices canbe provided for this purpose.

In one embodiment of the present invention, a result tree, a so-called“scoring tree”, is generated. Said result tree is generated on the basisof data from results in the game and the course of the game. The scoringtree is created by means of a computer and stored on a suitable storagemedium. In this case, suitable storage media comprise internal andexternal storage means, such as, for example, cloud storage means,network storage means and/or hard disk storage means. Each eventgenerates in the scoring tree a corresponding data set defining theevent. On the basis of this event and preferably with consultation ofthe conditions, the potential further course of the game is ascertained.The latter can be displayed on devices suitable therefor, such assmartphones, screens, computers, laptops, tablets and similar devices.After the match has ended, the scoring tree thus contains all potentialscores at every point in time in the game, wherein each score has aprobability that describes its plausibility. A check for thecompatibility of all possible scores with the detected events iseffected. This involves checking whether a score is actually possibleagainst the background of the detected events. FIG. 4 shows by way ofexample on the basis of a course of the game in tennis how firstly theevents present in the scoring tree are checked for compatibility withthe event “change of ends” and then the plausibility of the remainingscores is calculated on the basis of the remaining probability. In thiscase, plausibility describes the coherence or correctness of thestatements and serves as a criterion for assessment between the realvalues ascertained and the calculated values. Through the course of thematch and the events covering therein, in one embodiment, thepotentially possible scores are delimited in real time and the result ofthe end of the game is corrected accordingly. In the case of events withregard to the change of ends in tennis, for example, only an odd numberof ended games is possible, such as 0:1, 2:1, 0:3 etc. In the case of achange of serving end, by contrast, for example only an even number ofended games is possible, such as 0:2, 1:1, 4:0 etc. In one preferredembodiment of the present invention, potential scores which do not meetthe conditions of a real event on the playing field/court are thereforedeleted from the scoring tree. Corresponding further potential scoresare adapted in the ratio of the probabilities after the process ofdeletion, such that the total probability gives 100%. The advantage ofthis method is that the result of the game is no longer dependent on theindividual decisions at the level of points, but rather can be correctedafterward by means of the key events. In one particularly preferredembodiment, the course of the game is registered as “incorrect” by thesystem only if the majority of the individual decisions is incorrect. Amajority is more than 50% of the decisions, preferably more than 60% ofthe decisions, particularly preferably more than 70%.

In one embodiment, with regard to recognizing the field/court, the atleast one person, the at least one object, the trajectories and theevents on the playing field/court and also backward scoring, theautomatic tracking method captures point-accurate scores for whichassociated statistics such as, for example, error, winner, ace or doublefault can be recognized and associated with the player position and thetype of stroke. In one embodiment of the invention, the method comprisescommands, the following steps: a) determining an event A; b) correlatingthe event with the player position and/or with the type of stroke; c)communicating the data to an imaging device.

In one embodiment of the present invention, all image information dataare extracted from just one video image, that is to say that each imageinformation point is recorded by a single camera. A three-dimensionalmodel is not necessary for this. In order to be able to reliablyrecognize the diverse events, such as strokes in tennis, for example,different images and/or perspectives are usually necessary in order todetermine a specific type, such as forehand, backhand, serve, volley,topspin, flat or slice. In order to circumvent this necessity, oneembodiment involves creating a database with images of alreadycategorized events, such as in tennis, for example, of tennis strokesfrom a camera with a predetermined position. The resulting pixels anddata with specified positioning are categorized with the associatedstrokes in a next step. The data generated therefrom are used to createa so-called convolutional neural network (CNN), i.e. an artificialneural network. In the case of this CNN, the input into the network iseffected by way of pixels which are stored in a specific point in aspecific neuron. These artificial generated neurons are interconnectedto form an artificial neural network, such that they can exchangemessages among one another. The connections of the neurons and/ornetworks have a numerical weighting that is adapted during the coiningprocess, such that a correctly trained network reacts correctly in thecase of an image or pattern to be recognized; in this respect, see forexample Bengio, Y & Lecun, Yann. (1997), Convolutional Networks forImages, Speech, and Time-Series, as already described above.

With the aid of the generated data and the simple recognition with theaid of just one image or one video from a single camera position,networked playing fields/courts can be produced within a very short timeand allow the player to acquire match statistics, point-by-point videoanalyses and/or personal coaching on the basis of the actual course ofthe game. In order to represent the correct course of the game, asexplained above, decisions are dependent on the behavior of the playerand the associated object, or ball, and are not merely calculated by asystem. The system described is therefore able to generate correct,point-based statistics, even if the decisions of player and systeminitially differ.

In one embodiment of the present invention, the generated data aretransmitted to one or more devices, which can comprise, but are notrestricted to, smartphones, tablets, screens, monitors, televisions,and/or computers. In order to provide individual training, in onepreferred embodiment of the present invention, there are situated ateach playing field/court in each case codes specific therefor, which canbe scanned for example with the aid of a smartphone or an associatedapplication (App) installed thereon. With this the player can log on tothe field/court, use the latter and cause his/her data to be analyzed.For this purpose, in one embodiment, after the match the data areuploaded to an external storage means, in which the player can view saiddata. Said storage means can be a cloud-based service, a network, asmartphone, a computer or similar devices or combinations that aresuitable for storing and reproducing the data.

The data for the method demonstrated above are generated with the aid ofa device comprising an image recording system. Said image recordingsystem consists of at least one camera arranged in such a way that theentire playing field/court is captured. As a result, image data from theentire playing field/court can be generated and determined. In onepreferred embodiment, the image data are extracted from two cameras,wherein one camera is directed at a first side of the playingfield/court, for example at the left side of the playing field/court,and the second camera is directed at a second side of the playingfield/court, for example the right-hand half of the court.

The at least one camera is mounted at a predetermined location. Thelocation is preferably the net posts in which the camera is arranged inan integrated manner. In a further embodiment, a camera, such as anetwork camera, for example, can additionally be provided. Said cameracan be arranged behind the baseline, for example. This additional videomaterial can be acquired for later analyses by player and trainer.

The data of the at least one camera are communicated to a computingmodule and/or a storage means by means of a transmission module. Thetransmission module can be a wired or wireless connection. In onepreferred embodiment, a wired variant is involved.

In this case, the computing unit comprises modules that perform steps ofthe method explained above. The computing unit therefore comprises atleast one module for playing field/court ascertainment, at least onemodule for person recognition, at least one module for objectdetermination, at least one module for trajectory recognition, at leastone module for event identification, and at least one module forevent-based result ascertainment.

In this case, the at least one module for playing field/courtascertainment determines the position, size and dimensions of theplaying field/court. For this purpose, firstly image data of at leastone camera are transmitted to the module by a transmission unit. In afurther step, a calibration is carried out within the field/courtrecognition module, wherein individual points in a two-dimensional (2D)image are assigned to a specific point in a two-dimensional (2D) modelof the playing field/court, as can be gathered from FIG. 1, for example.The calibration is preferably effected by way of a homography matrix inwhich at least four points are used which correlate in the image and themodel, these particularly preferably being four corner points of theplaying field/court.

The at least one module for recognizing persons on the playingfield/court determines players or other objects on the playingfield/court, as already described with regard to the method. For thispurpose, firstly image data of at least one camera are transmitted tothe module by a transmission unit. A further step involves generating atleast one background image of an empty playing field/court. Thefollowing step involves creating a background model. For this purpose,firstly at least one image of the empty playing field/court is recorded.In the further step, using this background image, i.e. the empty playingfield/court, initialization is effected using the known “Mixture ofGaussians Background Subtractor” and a background model is calculated,which is regularly updated over the duration of the run time of themethod. Said subtractor is used to separate objects in the foregroundand background of an image and to carry out a check with regard to theirshape. The separation in each newly recorded image is effected byforming the absolute difference with the background model. The generateddata are used to create a foreground mask. In one preferred embodiment,it has the property that all pixels belonging to the foreground have thevalue 255 and the pixels of the background have the value 0, whereinpreferably pixels for which the difference is greater than a definedlimit value are determined as foreground and pixels with a smallerdifference are determined as background.

The at least one module for recognizing objects, such as balls, forexample, on the playing field/court identifies movements on eachrecorded image. In this case, firstly image data from a camera have tobe communicated to the object recognition module via the transmissionmodule or a previous module. The following steps involve firstlycreating difference images in respect of successive images andcalculating therefrom absolute difference images, in which a movement isvisible at two places in the difference image. In one preferredembodiment, the two places are a first position, at which the ball wassituated in the previous image, and a second position, at which the ballis situated in the current image. This is followed by generating a firstmovement mask for the one first difference image and a second movementmask for the one second difference image. These masks are used forascertaining a final movement mask that combines both movements. Theindividual pixels of both movement masks are preferably added by abitwise AND operator.

The module for trajectory recognition, like the modules before, usesimage data from at least one camera which were communicated to thismodule by means of a previous module or the transmission module. In thiscase, firstly potential starting points of a trajectory are defined.They are combined with all potential objects of the temporally followingimage and a check is made for each object as to whether an alreadyexisting trajectory is continued. These data are taken as a basis forascertaining positions of the object on successive images. In thesubsequent step, the two trajectories are compared with one another andthe trajectory course is determined. The trajectory is classified on thebasis of said trajectory course, preferably taking account of the nexttrajectory of the object that follows the trajectory. In one preferredembodiment, pixels of objects which correlate with pixels of persons arenot taken into account.

The module for identifying events in the course of the game compares theat least one position of at least one person and one object with theplaying field/court model. For this purpose, the module communicateswith one of the previous modules and/or a storage means to which saidone of the preceding modules has transmitted data ascertained and whichhas stored the latter. On the basis of these data, the module foridentifying events correlates them with databases in order to identifythe event. For this purpose, the module can communicate with amultiplicity of databases or information storage means to which it isconnected. In one embodiment, artificial intelligences trained inadvance are additionally used to classify the events. In one preferredembodiment, said intelligences are a convolutional neural network (CNN).

The module for event-oriented result capture in the present devicecorrects the score calculated in the course of the game by means ofchecking the possibilities of all possible scores with the real eventsrecognized by the previous modules. For this purpose, the modules areconnected to one another directly or indirectly and/or the acquisitionof data for the required information is effected by way of a storagemeans that saves all generated data from one of the previous modules. Inone preferred embodiment, potential scores are adapted in the ratio ofthe probabilities, such that the total probability gives 100%.

The above-described method or at least part of this method for objecttracking on a playing field/court comprising field/court recognition,object recognition, trajectory recognition, event recognition andbackward scoring is implemented on the computing module, such as acomputer, for example. Therefore, the present invention furthermorerelates to a computer program for carrying out a method described abovewhen the computer program is executed on a computer.

When the method is implemented, at least some of the steps belonging tothe method are carried out by a processor by the execution ofinstructions. In a further embodiment, instructions or some of theinstructions for carrying out the method described and/or forimplementing the method described in a system can be stored on anon-transitory, computer-readable data carrier.

The device of the present invention can comprise a multiplicity ofidentical and/or different modules. “Components” or “functional units”are also referred to as modules. Furthermore, modules and/or componentscan also be “computer-executed” and/or “computer-implemented”. In thiscase, the modules are implemented in the context of a computer systemthat typically comprises a processor and a storage means. In general, amodule is a component of a system that performs specific operations forthe implementation of a specific functionality. Examples offunctionalities include the reception of measured values (such as imagedata, for example) or the calculation of the field/court by means of acalculation module. However, the modules can also have furtherfunctionalities described in the above embodiments concerning themethod.

The term “module” here encompasses a tangible entity which is physicallyconstructed, is permanently configured (e.g. hardwired) or istemporarily configured (e.g. programmed) to operate in a specific way orto perform specific operations described herein. In embodiments in whichthe modules are temporarily configured (e.g. programmed), not everymodule need be configured or instantiated at every point in time. By wayof example, a general processor can be configured in such a way that itimplements different modules at varying times. In some embodiments, aprocessor implements a module by executing instructions which implementat least part of the functionality of the module. Optionally, a storagemeans can store the instructions (e.g. as computer code) which are readby the processor and processed and have the effect that the processorperforms at least some operations involved in the implementation of thefunctionality of the module.

Additionally or alternatively, in one embodiment, a storage means thatcan comprise one or more storage devices can store data which are readand processed by the processor in order to implement at least part ofthe functionality of the module. In a further embodiment, the storagemeans can comprise one or more hardware elements which can storeinformation that is accessible for a processor. In one embodiment, thestorage means can be situated at least partly as a part of the processoror on the same chip as the processor and/or can be a material elementseparate from the processor.

In one embodiment, the at least one processor executes instructionswhich are stored on the storage means and which perform operationsinvolved in the implementation of the functionality of a specificmodule. The at least one processor can additionally operate in such away that the performance of the relevant operations is supported in anenvironment with “Cloud Computing” or as “Software-as-a-Service” (SaaS).By way of example, at least some of the operations involved in theimplementation of a module can be performed by a group of computerswhich are accessible via a network such as, for example, the Internetand/or via one or more corresponding interfaces, such as applicationprogramming interfaces (API). Optionally, some of the modules can beimplemented in a distributed manner among a plurality of processors. Inone embodiment, the at least one processor can be situated at onegeographical location or can be distributed among a plurality ofgeographical locations. Optionally, some modules can comprise theexecution of instructions on devices which belong to the users and/orare situated alongside the players or spectators. By way of example,methods which comprise a presentation of results, for example, can beimplemented partly or completely on processors that belong to devices ofthe players. Said devices are for example laptops, tablets, smartphones,but this listing is not limited to the devices listed, but rather canencompass any known device. Furthermore, in one embodiment, data can beuploaded to cloud-based servers. In some embodiments, modules canprovide information to other modules and/or receive information fromother modules. Accordingly, such modules can be regarded ascommunicatively coupled. If a plurality of such modules are presentsimultaneously, communications can be achieved by means of signaltransmission. In embodiments in which modules are configured orinstantiated at different times, communications between such modules canbe achieved for example by the storage and retrieval of information instorage structures that can be accessed by a plurality of modules. Inone embodiment, one module can perform an operation and store the outputof this operation on a storage device to which it is communicativelycoupled. Another module can then access the storage device at a laterpoint in time in order to retrieve and process the stored output.

In this connection, the present invention furthermore relates to acomputer program product comprising a storage medium, on which is storeda computer program that comprises the above-described method for gamemonitoring comprising field/court recognition, person recognition,object recognition, trajectory recognition, event recognition andbackward scoring. In one preferred embodiment, the computer product isloaded directly into the internal memory of a digital computer andcomprises software code sections that perform the steps of gamemonitoring when the product runs on a computer. In this case, the termcomputer program product encompasses a computer program stored on acarrier, such as, for example, RAM, ROM, CD, apparatuses and similardevices; an embedded system as a comprehensive system comprising acomputer program, such as, for example, an electronic device comprisinga computer program; a network of computer-implemented computer programssuch as, for example, server systems, client systems, cloud computingsystems and the like; and/or computers on which a computer program isloaded, runs, is stored, is executed or is developed.

Although the methods disclosed herein can be described and illustratedwith regard to specific steps carried out in a specific order, it shouldbe assumed that these steps can be combined, subdivided and/orrearranged in order to form an equivalent method without departing fromthe teachings of the embodiments. Accordingly, unless expresslyspecified herein, the order and grouping of the steps do not constituteany restriction of the embodiments. Furthermore, the methods andmechanisms of the embodiments are described in the singular form in somecases for reasons of clarity. However, some embodiments, unlessindicated otherwise, can comprise a plurality of iterations of a methodor a plurality of instantiations of a mechanism. If one processor isdisclosed in one embodiment, for example, the scope of application ofthe embodiment is also intended to cover the use of a plurality ofprocessors. Specific features of the embodiments which have possiblybeen described in the context of separate embodiments for reasons ofclarity can additionally be provided in different combinations in asingle embodiment. Conversely, different features of the embodimentswhich have possibly been described in the context of a single embodimentfor reasons of space can additionally be provided separately or in anysuitable subcombination.

In a further embodiment, the methods and programs can be implementedwith different computer system configurations. These computer systemsencompass but are not restricted to cloud computing, client servermodels, grid computing, peer-to-peer, handheld devices, multiprocessorsystems, microprocessor-based systems, programmable consumerelectronics, mini computers and/or mainframes. Additionally oralternatively, some of the embodiments can be implemented in adistributed computer environment in which the tasks are carried out byremote processing devices connected via a communication network. In adistributed computer environment, program components can be localized inboth local and remote computers and/or storage devices. Additionally oralternatively, some of the embodiments can be implemented in the form ofa service such as Infrastructure-as-a-Service (IaaS),Platform-as-a-Service (PaaS), Software-as-a-Service (SaaS) and/orNetwork-as-a-Service (NaaS).

These and other embodiments of the present invention are disclosed inthe description and the examples and are encompassed thereby. Thefeatures mentioned in the above description and in the claims arefurthermore combinable in any desired selection and the featurecombinations that are expedient in the context of the invention shouldbe deemed to be disclosed. Further literature concerning any knownmaterials, methods and applications that can be used in accordance withthe present invention can be retrieved from public libraries anddatabases, for example using electronic devices. A more completeunderstanding of the invention can be obtained by reference to thefigures and examples, which have been provided for illustration purposesand are not intended to restrict the scope of the invention.

EXAMPLES Example Program Sequence for Backward Scoring on the Basis ofthe Example of Tennis

FIG. 3 illustrates a program sequence for backward scoring on the basisof the example of a game of tennis. This involves firstly providing alist with detected events (101), which list was generated beforehand bydata ascertainment. In the further step (102), an empty scoring tree isinitialized. In the latter, the individual events are processed (103).There follows the determination of a rally (104). If a rally has takenplace, then the following step (105) involves determining all potentialscores which have resulted from this rally. The scoring tree is updatedaccording to the corresponding score and new “leaves” are added (106).

If no rally has taken place, it is determined whether an end of therally can be determined (107). If an end has occurred, the probabilityfor the outcome of the rally is calculated in the following step (108).The calculated probability is noted (109) at the leaves of the scoringtree and the total probability for each score is updated (110).

If no rally has taken place, it is determined whether an end of the gamehas taken place (111). If an end has been detected, then all scores withincomplete games are removed from the scoring tree (112). In the nextstep, the change of ends is identified (113). If a change of ends hastaken place, then all scores with an even number of games are removedfrom the scoring tree (114). However, if no change of ends is detected,all scores with an odd number of games are removed from the scoring tree(115).

If an end of the game has not been determined, a check is made as towhether the match has finished (116). If this is the case, all scoreswith incomplete games (117) and sets (118) are removed from the scoringtree. The last step involves determining (119) the score with thehighest probability as the final score and outputting (120) it as thescore.

1. A method for game monitoring comprising the following steps: a)determining the playing field/court; b) recognizing at least one personon the playing field/court; c) identifying objects on the playingfield/court; d) determining at least one trajectory; wherein at leastone event is identified by means of which the scores calculated in thecourse of the game are corrected by means of backward scoring and thereal final score of the game is output.
 2. The method as claimed inclaim 1, wherein determining the playing field/court is effected bymeans of calibration, wherein individual points in a two-dimensional(2D) image are assigned to a specific point in a two-dimensional (2D)model of the playing field/court, wherein the calibration is preferablyeffected by way of a homography matrix in which at least four points areused which correlate in the image and the model, these particularlypreferably being four corner points of the playing field/court.
 3. Themethod as claimed in claim 2, wherein recognizing persons on the playingfield/court comprises the following steps: a) generating at least onebackground image of an empty playing field/court; b) generating abackground model; c) separating each background image from eachcurrently recorded image which comprises a person; d) determining anabsolute difference between the background model and the currentlyrecorded image, and e) generating a foreground mask, wherein theforeground mask is characterized in that all pixels belonging to theforeground have the value 255 and the pixels of the background have thevalue 0, wherein preferably pixels for which the difference is greaterthan a defined limit value are determined as foreground and pixels witha smaller difference are determined as background.
 4. The method asclaimed in claim 3, wherein the object on the playing field/court isidentified by means of movements of each recorded image, whereindetermining the movement preferably comprises the following steps: a)creating difference images in respect of successive images; b)calculating absolute difference images, in which a movement is visibleat two places in the difference image, it preferably being a firstposition, at which the ball was situated in the previous image, and asecond position, at which the ball is situated in the current image; c)generating a first movement mask for the one first difference image anda second movement mask for the one second difference image; and d)generating a final movement mask comprising the movements in the onefirst movement mask and the one second movement mask; the individualpixels of both movement masks are preferably added by a bitwise ANDoperator.
 5. The method as claimed in claim 4, wherein the trajectorypath or the trajectory of the object is ascertained, comprising thefollowing steps: a) fixing a first and a second starting point of apotential trajectory; b) combining the potentially fixed starting pointswith all potential objects of the temporally following image; c)checking each object as to whether an already existing trajectory iscontinued; d) ascertaining positions of the object on successive images;e) comparing the trajectories from a) and d); f) determining atrajectory course; and g) classifying the trajectory by determining asecond trajectory following the first trajectory, wherein pixels ofobjects which correlate with pixels of persons are preferably not takeninto account.
 6. The method as claimed in claim 5, wherein identifyingevents in the course of the game comprises the following steps: a)capturing a triggering event; b) comparing the position of at least oneperson and one object with the playing field/court model; c) correlatingwith a database, wherein the events have preferably been captured in acategorized data set in a previous step in order to carry out thecorrelating with the aid of an artificial intelligence trained with thisdata set.
 7. The method as claimed in claim 6, wherein correcting thescore calculated in the course of the game comprises the followingsteps: a) monitoring the possible scores during the game; b) checkingthe compatibility of all possible scores with the detected events; c)checking the plausibility of the remaining scores.
 8. A device fortracking at least one object and player on a playing field/courtcomprising: a) an image recording system comprising i) at least onecamera arranged in such a way that the entire playing field/court iscaptured; ii) at least one transmission module designed to store thedata recorded by the camera and/or to transmit them to a computing unit,a storage means, or system; wherein the at least one camera and the atleast one transmission module are attached to at least one net post; b)a computing unit, comprising modules for iii) playing field/courtascertainment; iv) person recognition; v) object determination; vi)trajectory recognition; vii) event identification; and viii) event-basedresult ascertainment.
 9. (canceled)
 10. (canceled)